Ml Debugging

Collecting both my ideas and resources from around the web on ML Debugging tools.
Debugging is
- Show me what's going on: visibility into the system.
  - Ideally: cheap, always available, very reliable
- Let me tweak something and tell me if it fixed the problem.
  - Ideally: fast, doesn't waste resources or time and proceeds from where the tweak was applied
Tactics
- Overfit a small batch and test that loss is almost negligible: https://twitter.com/rasbt/status/1555286200004534273
Tools
- Tensorboard
- Weights & Biases
- AIM stack
- Weight watcher: https://pypi.org/project/weightwatcher/
  - https://calculatedcontent.com/
Resources
- ICLR on debugging (2019) ICLR
- Amazon Sagemaker Debugger